1 Review

We have known for linear model $$\begin{array}{}\text{(1)}& y_t={\beta }_0+{\beta }_{1}t+{\epsilon }_t,\end{array}$$ where ${\displaystyle t=1,\cdots ,n}$, ${\displaystyle {\epsilon }_t\sim \mathcal{N}(0,{\sigma }^2)}$, ${\displaystyle {\beta }_0,{\beta }_1,\log \sigma \sim \mathrm{Unif}(-C,C)}$, the posterior is $$f_{{\beta }_0,{\beta }_1|\text{data}}({\beta }_0,{\beta }_1)\propto {\left(\frac{S({\hat{\beta }}_0,{\hat{\beta }}_1)}{S({\beta }_0,{\beta }_1)}\right)}^{\frac{n}{2}},$$ where ${\displaystyle S({\beta }_0,{\beta }_1)=\sum _{t=1}^n(y_i-{\beta }_0-{\beta }_{1}t{)}^2}$.

For $$\begin{array}{}\text{(2)}& y_t={\beta }_0+{\beta }_{1}t+{\beta }_2{t}^2+{\beta }_3{t}^3+{\epsilon }_t,\end{array}$$ similarly we have $$f_{{\beta }_0,{\beta }_1,{\beta }_2,{\beta }_3|\mathrm{data}}({\beta }_0,{\beta }_1,{\beta }_2,{\beta }_3)\propto {\left[\frac{S({\hat{\beta }}_0,{\hat{\beta }}_1,{\hat{\beta }}_2,{\hat{\beta }}_3)}{S({\beta }_0,{\beta }_1,{\beta }_2,{\beta }_3)}\right]}^{\frac{n}{2}},$$ where ${\displaystyle S({\beta }_0,{\beta }_1,{\beta }_2,{\beta }_3)=\sum _{t=1}^n(y_t-{\beta }_0-{\beta }_{1}t-{\beta }_2{t}^2-{\beta }_3{t}^3{)}^2}$.

2 Vector Matrix Notation

Denote $$y_{n\times 1}=\left(\begin{array}{c}{y}_1\\ ⋮\\ y_n\end{array}\right),\beta =\left(\begin{array}{c}{\beta }_0\\ {\beta }_1\end{array}\right),X=\left[\begin{array}{cc}1& 1\\ 1& 2\\ ⋮& ⋮\\ 1& n\end{array}\right]$$ for (1), or $$\beta =\left(\begin{array}{c}{\beta }_0\\ ⋮\\ {\beta }_3\end{array}\right),X=\left[\begin{array}{cccc}1& 1& 1& 1\\ ⋮& ⋮& ⋮& ⋮\\ 1& n& n^2& n^3\end{array}\right]$$ for (2).

2.1 Minimizer of Squared Loss

We can denote ${\displaystyle S(\beta )=||y-X\beta |{|}^2}$. Since $$\begin{array}{rl}S(\beta )=||y-X\beta |{|}^2& =(y-X\beta {)}^{\mathrm{T}}(y-X\beta )\\ & =y^{\mathrm{T}}y-y^{\mathrm{T}}X\beta -{\beta }^{\mathrm{T}}{X}^{\mathrm{T}}y+{\beta }^{\mathrm{T}}{X}^{\mathrm{T}}X\beta ,\\ \mathrm{\nabla }S(\beta )& =-X^{\mathrm{T}}y-X^{\mathrm{T}}y+2X^{\mathrm{T}}X\beta =2X^{\mathrm{T}}(X\beta -y).\end{array}$$ Solve for ${\displaystyle \mathrm{\nabla }S(\beta )=0}$, we have ${\displaystyle \hat{\beta }=(X^{\mathrm{T}}X{)}^{-1}{X}^{\mathrm{T}}y}$, which minimizes ${\displaystyle S(\beta )}$.

2.2 Pythagorean Identity

The cross term vanishes because of ${\displaystyle \hat{\beta }}$.

Now we go back to $$\begin{array}{rl}{f}_{\beta |\mathrm{data}}(\beta )& \propto {\left(\frac{S(\hat{\beta })}{S(\beta )}\right)}^{\frac{n}{2}}={\left[\frac{S(\hat{\beta })}{S(\hat{\beta })+(\hat{\beta }-\beta {)}^{\mathrm{T}}{X}^{\mathrm{T}}X(\hat{\beta }-\beta )}\right]}^{\frac{n}{2}}\\ & ={[1+(\hat{\beta }-\beta {)}^{\mathrm{T}}\frac{X^{\mathrm{T}}X}{S(\hat{\beta })}(\hat{\beta }-\beta )]}^{-\frac{n}{2}}.\end{array}$$

3 Multivariate Normal & t-Distribution

Suppose ${\displaystyle X\sim {\mathcal{N}}_p(\mu ,\mathrm{\Sigma })}$, ${\displaystyle V\sim {\chi }_k^2}$, denote $$T=\mu +\frac{X-\mu }{\sqrt{V/k}}=\left(\begin{array}{c}{\mu }_1+\frac{X_1-{\mu }_1}{\sqrt{V/k}}\\ ⋮\\ {\mu }_k+\frac{X_k-{\mu }_k}{\sqrt{V/k}}\end{array}\right)\sim t_{p,k}(\mu ,\mathrm{\Sigma })$$ is defined as t-distribution.

Note when ${\displaystyle k}$ is large, ${\displaystyle t_{p,k}(\mu ,\mathrm{\Sigma })}$ is close to ${\displaystyle {\mathcal{N}}_p(\mu ,\mathrm{\Sigma })}$.

4 Back to Bayesian Inference

Therefore for second model, ${\displaystyle p=4}$, the posterior is t-distribution: $$\beta |\mathrm{data}\sim t_{4,n-4}(\hat{\beta },\frac{S(\hat{\beta })}{n-4}(X^{\mathrm{T}}X{)}^{-1}).$$

In previous notes we have unbiased estimator ${\displaystyle \hat{\sigma }=\sqrt{\frac{S(\hat{\beta })}{n-4}}}$ (note ${\displaystyle p}$ has changed to ${\displaystyle 4}$ here), so $$\beta |\mathrm{data}\sim t_{4,n-4}(\hat{\beta },{\hat{\sigma }}^2(X^{\mathrm{T}}X{)}^{-1}).$$

When the degrees of freedom of the t-distribution is large, i.e. when ${\displaystyle n}$ is large, we can approximate the second model with $$\beta |\mathrm{data}\sim {\mathcal{N}}_4(\hat{\beta },{\hat{\sigma }}^2(X^{\mathrm{T}}X{)}^{-1}).$$

For general setting $$y_t={\beta }_0+{\beta }_1{x}_{t1}+{\beta }_2{x}_{t2}+\cdots +{\beta }_m{x}_{im}+{\epsilon }_i,{\epsilon }_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\sigma }^2),$$ ${\displaystyle p=m+1}$ (as ${\displaystyle \beta =({\beta }_0,\cdots ,{\beta }_m)}$ has ${\displaystyle m+1}$ components), so ${\displaystyle k=n-p=n-m-1}$. It's also clear that ${\displaystyle \mu =\hat{\beta }}$, and $$\frac{1}{k}{\mathrm{\Sigma }}^{-1}=\frac{X^{\mathrm{T}}X}{S(\hat{\beta })}\Rightarrow \mathrm{\Sigma }=\frac{S(\hat{\beta })}{k}(X^{\mathrm{T}}X{)}^{-1}=\frac{S(\hat{\beta })}{n-m-1}(X^{\mathrm{T}}X{)}^{-1},$$ therefore $$\beta |\mathrm{data}\sim t_{n-m-1,m+1}(\hat{\beta },\frac{S(\hat{\beta })}{n-m-1}(X^{\mathrm{T}}X{)}^{-1}).$$ Similarly with ${\displaystyle p=4}$, denote $$\hat{\sigma }=\sqrt{\frac{S(\hat{\beta })}{n-m-1}},$$ so $$\begin{array}{}\text{(3.1)}& \beta |\mathrm{data}\sim t_{n-m-1,m+1}(\hat{\beta },{\hat{\sigma }}^2(X^{\mathrm{T}}X{)}^{-1}).\end{array}$$ By fact, $$\begin{array}{}\text{(3.2)}& {\beta }_j|\mathrm{data}\sim t_{n-m-1,1}({\hat{\beta }}_j,{\hat{\sigma }}^2(X^{\mathrm{T}}X{)}^{j+1,j+1}).\end{array}$$
When ${\displaystyle n}$ is large, (3.1) goes to ${\displaystyle {\mathcal{N}}_{m+1}(\hat{\beta },{\hat{\sigma }}^2(X^{\mathrm{T}}X{)}^{-1})}$, (3.2) goes to ${\displaystyle \mathcal{N}({\hat{\beta }}_j,{\hat{\sigma }}^2(X^{\mathrm{T}}X{)}^{j+1,j+1})}$.